{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "intro_to_sparse_data_and_embeddings.ipynb",
      "version": "0.3.2",
      "views": {},
      "default_view": {},
      "provenance": [],
      "collapsed_sections": [
        "mNCLhxsXyOIS",
        "eQS5KQzBybTY",
        "copyright-notice"
      ]
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "#### Copyright 2017 Google LLC."
      ],
      "metadata": {
        "colab_type": "text",
        "id": "copyright-notice"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ],
      "metadata": {
        "colab": {
          "autoexec": {
            "wait_interval": 0,
            "startup": false
          }
        },
        "cellView": "both",
        "colab_type": "code",
        "id": "copyright-notice2"
      },
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "PTaAdgy3LS8W",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " # \ud76c\uc18c \ub370\uc774\ud130 \ubc0f \uc784\ubca0\ub529 \uc18c\uac1c\n",
        "\n",
        "**\ud559\uc2b5 \ubaa9\ud45c:**\n",
        "* \uc601\ud654 \ub9ac\ubdf0 \ubb38\uc790\uc5f4 \ub370\uc774\ud130\ub97c \ud76c\uc18c \ud2b9\uc131 \ubca1\ud130\ub85c \ubcc0\ud658\ud55c\ub2e4\n",
        "* \ud76c\uc18c \ud2b9\uc131 \ubca1\ud130\ub97c \uc0ac\uc6a9\ud558\uc5ec \uac10\uc815 \ubd84\uc11d \uc120\ud615 \ubaa8\ub378\uc744 \uad6c\ud604\ud55c\ub2e4\n",
        "* \ub370\uc774\ud130\ub97c \ub450 \ucc28\uc6d0\uc73c\ub85c \ud22c\uc601\ud558\ub294 \uc784\ubca0\ub529\uc744 \uc0ac\uc6a9\ud558\uc5ec \uac10\uc815 \ubd84\uc11d DNN \ubaa8\ub378\uc744 \uad6c\ud604\ud55c\ub2e4\n",
        "* \uc784\ubca0\ub529\uc744 \uc2dc\uac01\ud654\ud558\uc5ec \ub2e8\uc5b4 \uac04\uc758 \uad00\uacc4\uc5d0 \ub300\ud574 \ubaa8\ub378\uc774 \ud559\uc2b5\ud55c \ub0b4\uc6a9\uc744 \ud655\uc778\ud55c\ub2e4\n",
        "\n",
        "\uc774 \uc2e4\uc2b5\uc5d0\uc11c\ub294 \ud76c\uc18c \ub370\uc774\ud130\uc5d0 \ub300\ud574 \uc54c\uc544\ubcf4\uace0 [ACL 2011 IMDB \ub370\uc774\ud130 \uc138\ud2b8](http://ai.stanford.edu/~amaas/data/sentiment/)\uc5d0\uc11c \uac00\uc838\uc628 \uc601\ud654 \ub9ac\ubdf0 \ud14d\uc2a4\ud2b8 \ub370\uc774\ud130\ub85c \uc784\ubca0\ub529\uc744 \uc0ac\uc6a9\ud574 \ubd05\ub2c8\ub2e4. \uc774 \ub370\uc774\ud130\ub294 \uc774\ubbf8 `tf.Example` \ud615\uc2dd\uc73c\ub85c \ucc98\ub9ac\ub418\uc5b4 \uc788\uc2b5\ub2c8\ub2e4.  "
      ]
    },
    {
      "metadata": {
        "id": "2AKGtmwNosU8",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " ## \uc124\uc815\n",
        "\n",
        "\uc6b0\uc120 \ud544\uc694\ud55c \ubaa8\ub4c8\uc744 import\ub85c \ubd88\ub7ec\uc624\uace0 \ud559\uc2b5 \ubc0f \ud14c\uc2a4\ud2b8 \ub370\uc774\ud130\ub97c \ub2e4\uc6b4\ub85c\ub4dc\ud569\ub2c8\ub2e4. [`tf.keras`](https://www.tensorflow.org/api_docs/python/tf/keras)\uc5d0 \ud3ec\ud568\ub41c \ud30c\uc77c \ub2e4\uc6b4\ub85c\ub4dc \ubc0f \uce90\uc2f1 \ub3c4\uad6c\ub97c \uc0ac\uc6a9\ud558\uc5ec \ub370\uc774\ud130 \uc138\ud2b8\ub97c \uac80\uc0c9\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4."
      ]
    },
    {
      "metadata": {
        "id": "jGWqDqFFL_NZ",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "from __future__ import print_function\n",
        "\n",
        "import collections\n",
        "import io\n",
        "import math\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import tensorflow as tf\n",
        "from IPython import display\n",
        "from sklearn import metrics\n",
        "\n",
        "tf.logging.set_verbosity(tf.logging.ERROR)\n",
        "train_url = 'https://download.mlcc.google.com/mledu-datasets/sparse-data-embedding/train.tfrecord'\n",
        "train_path = tf.keras.utils.get_file(train_url.split('/')[-1], train_url)\n",
        "test_url = 'https://download.mlcc.google.com/mledu-datasets/sparse-data-embedding/test.tfrecord'\n",
        "test_path = tf.keras.utils.get_file(test_url.split('/')[-1], test_url)"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "6W7aZ9qspZVj",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " ## \uac10\uc815 \ubd84\uc11d \ubaa8\ub378 \ub9cc\ub4e4\uae30"
      ]
    },
    {
      "metadata": {
        "id": "jieA0k_NLS8a",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " \uc774 \ub370\uc774\ud130\ub85c \uac10\uc815 \ubd84\uc11d \ubaa8\ub378\uc744 \ud559\uc2b5\uc2dc\ucf1c \ub9ac\ubdf0\uac00 \uc804\ubc18\uc801\uc73c\ub85c *\uae0d\uc815\uc801*(\ub77c\ubca8 1)\uc778\uc9c0 \uc544\ub2c8\uba74 *\ubd80\uc815\uc801*(\ub77c\ubca8 0)\uc778\uc9c0\ub97c \uc608\uce21\ud574 \ubcf4\uaca0\uc2b5\ub2c8\ub2e4.\n",
        "\n",
        "\uc774\ub807\uac8c \ud558\ub824\uba74 \ubb38\uc790\uc5f4 \uac12\uc778 `\ub2e8\uc5b4`\ub97c *\uc5b4\ud718*, \uc989 \ub370\uc774\ud130\uc5d0 \ub098\uc62c \uac83\uc73c\ub85c \uc608\uc0c1\ub418\ub294 \uac01 \ub2e8\uc5b4\uc758 \ubaa9\ub85d\uc744 \uc0ac\uc6a9\ud558\uc5ec \ud2b9\uc131 \ubca1\ud130\ub85c \ubcc0\ud658\ud569\ub2c8\ub2e4. \uc774 \uc2e4\uc2b5\uc744 \uc9c4\ud589\ud558\uae30 \uc704\ud574 \uc81c\ud55c\ub41c \ub2e8\uc5b4 \uc9d1\ud569\uc744 \uac16\ub294 \uc18c\uaddc\ubaa8 \uc5b4\ud718\ub97c \ub9cc\ub4e4\uc5c8\uc2b5\ub2c8\ub2e4. \uc774\ub7ec\ud55c \ub2e8\uc5b4\ub294 \ub300\ubd80\ubd84 *\uae0d\uc815* \ub610\ub294 *\ubd80\uc815*\uc744 \uac15\ud558\uac8c \uc554\uc2dc\ud558\ub294 \uac83\uc774 \ubc1d\ud600\uc84c\uc9c0\ub9cc \uc77c\ubd80\ubd84\uc740 \ub2e8\uc21c\ud788 \ud765\ubbf8\ub97c \uc704\ud574 \ucd94\uac00\ub418\uc5c8\uc2b5\ub2c8\ub2e4.\n",
        "\n",
        "\uc5b4\ud718\uc758 \uac01 \ub2e8\uc5b4\ub294 \ud2b9\uc131 \ubca1\ud130\uc758 \uc88c\ud45c\uc5d0 \ub9e4\ud551\ub429\ub2c8\ub2e4. \uc608\uc758 \ubb38\uc790\uc5f4 \uac12\uc778 `\ub2e8\uc5b4`\ub97c \uc774 \ud2b9\uc131 \ubca1\ud130\ub85c \ubcc0\ud658\ud558\uae30 \uc704\ud574, \uc608 \ubb38\uc790\uc5f4\uc5d0 \uc5b4\ud718 \ub2e8\uc5b4\uac00 \ub098\uc624\uc9c0 \uc54a\uc73c\uba74 \uac01 \uc88c\ud45c\uc758 \uac12\uc5d0 0\uc744 \uc785\ub825\ud558\uace0 \uc5b4\ud718 \ub2e8\uc5b4\uac00 \ub098\uc624\uba74 1\uc744 \uc785\ub825\ud558\ub3c4\ub85d \uc778\ucf54\ub529\ud558\uaca0\uc2b5\ub2c8\ub2e4. \uc608\uc758 \ub2e8\uc5b4 \uc911 \uc5b4\ud718\uc5d0 \ub098\uc624\uc9c0 \uc54a\ub294 \ub2e8\uc5b4\ub294 \ubb34\uc2dc\ub429\ub2c8\ub2e4."
      ]
    },
    {
      "metadata": {
        "id": "2HSfklfnLS8b",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " **\ucc38\uace0:** *\ubb3c\ub860 \ub354 \ud070 \uc5b4\ud718\ub97c \uc0ac\uc6a9\ud560 \uc218\ub3c4 \uc788\uc73c\uba70 \uc774\ub7ec\ud55c \uc5b4\ud718\ub97c \ub9cc\ub4dc\ub294 \ub370 \ud2b9\ud654\ub41c \ub3c4\uad6c\ub4e4\uc774 \uc788\uc2b5\ub2c8\ub2e4. \ubfd0\ub9cc \uc544\ub2c8\ub77c \uc5b4\ud718\uc5d0 \ub098\uc624\uc9c0 \uc54a\ub294 \ub2e8\uc5b4\ub97c \ub2e8\uc21c\ud788 \ubb34\uc2dc\ud558\uc9c0 \uc54a\uace0 \uc801\uc740 \uc218\uc758 OOV(out-of-vocabulary) \ubc84\ud0b7\uc744 \ub3c4\uc785\ud558\uc5ec \ud574\ub2f9 \ub2e8\uc5b4\ub97c \ud574\uc2dc\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4. \uba85\uc2dc\uc801\uc778 \uc5b4\ud718\ub97c \ub9cc\ub4dc\ub294 \ub300\uc2e0 \uac01 \ub2e8\uc5b4\ub97c \ud574\uc2dc\ud558\ub294 __\ud2b9\uc131 \ud574\uc2f1__ \uc811\uadfc\ubc95\uc744 \uc0ac\uc6a9\ud560 \uc218\ub3c4 \uc788\uc2b5\ub2c8\ub2e4. \uc774 \ubc29\ubc95\uc740 \uc2e4\ubb34\uc5d0\ub294 \uc801\ud569\ud558\uc9c0\ub9cc \ud574\uc11d \uac00\ub2a5\uc131\uc774 \uc0ac\ub77c\uc9c0\ubbc0\ub85c \uc2e4\uc2b5 \ubaa9\uc801\uc73c\ub85c\ub294 \uc720\uc6a9\ud558\uc9c0 \uc54a\uc2b5\ub2c8\ub2e4. \uc774\uc640 \uad00\ub828\ub41c \ub3c4\uad6c\uc5d0 \ub300\ud574\uc11c\ub294 tf.feature_column \ubaa8\ub4c8\uc744 \ucc38\uc870\ud558\uc138\uc694.*"
      ]
    },
    {
      "metadata": {
        "id": "Uvoa2HyDtgqe",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " ## \uc785\ub825 \ud30c\uc774\ud504\ub77c\uc778 \uad6c\ucd95"
      ]
    },
    {
      "metadata": {
        "id": "O20vMEOurDol",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " \uc6b0\uc120 \ud150\uc11c\ud50c\ub85c\uc6b0 \ubaa8\ub378\ub85c \ub370\uc774\ud130\ub97c \uac00\uc838\uc624\ub294 \uc785\ub825 \ud30c\uc774\ud504\ub77c\uc778\uc744 \uad6c\uc131\ud558\uaca0\uc2b5\ub2c8\ub2e4. \ub2e4\uc74c \ud568\uc218\ub97c \uc0ac\uc6a9\ud558\uc5ec [TFRecord](https://www.tensorflow.org/guide/datasets#consuming_tfrecord_data) \ud615\uc2dd\uc778 \uc785\ub825 \ubc0f \ud14c\uc2a4\ud2b8 \ub370\uc774\ud130\ub97c \ud30c\uc2f1\ud558\uace0 \ud2b9\uc131\uacfc \ud574\ub2f9 \ub77c\ubca8\ub85c \uc774\ub8e8\uc5b4\uc9c4 dict\ub97c \ubc18\ud658\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4."
      ]
    },
    {
      "metadata": {
        "id": "SxxNIEniPq2z",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "def _parse_function(record):\n",
        "  \"\"\"Extracts features and labels.\n",
        "  \n",
        "  Args:\n",
        "    record: File path to a TFRecord file    \n",
        "  Returns:\n",
        "    A `tuple` `(labels, features)`:\n",
        "      features: A dict of tensors representing the features\n",
        "      labels: A tensor with the corresponding labels.\n",
        "  \"\"\"\n",
        "  features = {\n",
        "    \"terms\": tf.VarLenFeature(dtype=tf.string), # terms are strings of varying lengths\n",
        "    \"labels\": tf.FixedLenFeature(shape=[1], dtype=tf.float32) # labels are 0 or 1\n",
        "  }\n",
        "  \n",
        "  parsed_features = tf.parse_single_example(record, features)\n",
        "  \n",
        "  terms = parsed_features['terms'].values\n",
        "  labels = parsed_features['labels']\n",
        "\n",
        "  return  {'terms':terms}, labels"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "SXhTeeYMrp-l",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " \ud568\uc218\uac00 \uc815\uc0c1\uc801\uc73c\ub85c \uc791\ub3d9\ud558\ub294\uc9c0 \ud655\uc778\ud558\uae30 \uc704\ud574 \ud559\uc2b5 \ub370\uc774\ud130\uc5d0 \ub300\ud55c `TFRecordDataset`\ub97c \uc0dd\uc131\ud558\uace0 \uc704 \ud568\uc218\ub97c \uc0ac\uc6a9\ud558\uc5ec \ub370\uc774\ud130\ub97c \ud2b9\uc131 \ubc0f \ub77c\ubca8\uc5d0 \ub9e4\ud551\ud569\ub2c8\ub2e4."
      ]
    },
    {
      "metadata": {
        "id": "oF4YWXR0Omt0",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "# Create the Dataset object.\n",
        "ds = tf.data.TFRecordDataset(train_path)\n",
        "# Map features and labels with the parse function.\n",
        "ds = ds.map(_parse_function)\n",
        "\n",
        "ds"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "bUoMvK-9tVXP",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " \ub2e4\uc74c \uc140\uc744 \uc2e4\ud589\ud558\uc5ec \ud559\uc2b5 \ub370\uc774\ud130 \uc138\ud2b8\uc5d0\uc11c \uccab \uc608\ub97c \ud655\uc778\ud569\ub2c8\ub2e4."
      ]
    },
    {
      "metadata": {
        "id": "Z6QE2DWRUc4E",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "n = ds.make_one_shot_iterator().get_next()\n",
        "sess = tf.Session()\n",
        "sess.run(n)"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "jBU39UeFty9S",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " \uc774\uc81c \ud150\uc11c\ud50c\ub85c\uc6b0 \uc5d0\uc2a4\ud2f0\uba54\uc774\ud130 \uac1c\uccb4\uc758 `train()` \uba54\uc18c\ub4dc\uc5d0 \uc804\ub2ec\ud560 \uc218 \uc788\ub294 \uc815\uc2dd \uc785\ub825 \ud568\uc218\ub97c \ub9cc\ub4e4\uaca0\uc2b5\ub2c8\ub2e4."
      ]
    },
    {
      "metadata": {
        "id": "5_C5-ueNYIn_",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "# Create an input_fn that parses the tf.Examples from the given files,\n",
        "# and split them into features and targets.\n",
        "def _input_fn(input_filenames, num_epochs=None, shuffle=True):\n",
        "  \n",
        "  # Same code as above; create a dataset and map features and labels.\n",
        "  ds = tf.data.TFRecordDataset(input_filenames)\n",
        "  ds = ds.map(_parse_function)\n",
        "\n",
        "  if shuffle:\n",
        "    ds = ds.shuffle(10000)\n",
        "\n",
        "  # Our feature data is variable-length, so we pad and batch\n",
        "  # each field of the dataset structure to whatever size is necessary.     \n",
        "  ds = ds.padded_batch(25, ds.output_shapes)\n",
        "  \n",
        "  ds = ds.repeat(num_epochs)\n",
        "\n",
        "  \n",
        "  # Return the next batch of data.\n",
        "  features, labels = ds.make_one_shot_iterator().get_next()\n",
        "  return features, labels"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Y170tVlrLS8c",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " ## \uc791\uc5c5 1: \ud76c\uc18c \uc785\ub825 \ubc0f \uba85\uc2dc\uc801 \uc5b4\ud718\uc640 \ud568\uaed8 \uc120\ud615 \ubaa8\ub378 \uc0ac\uc6a9\n",
        "\n",
        "\uccab \ubc88\uc9f8 \ubaa8\ub378\ub85c\uc11c 50\uac1c\uc758 \uc815\ubcf4 \ub2e8\uc5b4\ub97c \uc0ac\uc6a9\ud558\uc5ec [`LinearClassifier`](https://www.tensorflow.org/api_docs/python/tf/estimator/LinearClassifier) \ubaa8\ub378\uc744 \ub9cc\ub4e4\uaca0\uc2b5\ub2c8\ub2e4. \ucc98\uc74c\uc5d0\ub294 \ub2e8\uc21c\ud558\uac8c \uc2dc\uc791\ud558\ub294 \uac83\uc774 \uc88b\uc2b5\ub2c8\ub2e4.\n",
        "\n",
        "\ub2e4\uc74c \ucf54\ub4dc\ub294 \ub2e8\uc5b4\uc5d0 \ub300\ud55c \ud2b9\uc131 \uc5f4\uc744 \ub9cc\ub4ed\ub2c8\ub2e4. [`categorical_column_with_vocabulary_list`](https://www.tensorflow.org/api_docs/python/tf/feature_column/categorical_column_with_vocabulary_list) \ud568\uc218\ub294 \ubb38\uc790\uc5f4\uacfc \ud2b9\uc131 \ubca1\ud130 \uac04\uc758 \ub9e4\ud551\uc744 \ud3ec\ud568\ud558\ub294 \ud2b9\uc131 \uc5f4\uc744 \ub9cc\ub4ed\ub2c8\ub2e4."
      ]
    },
    {
      "metadata": {
        "id": "B5gdxuWsvPcx",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "# 50 informative terms that compose our model vocabulary. \n",
        "informative_terms = (\"bad\", \"great\", \"best\", \"worst\", \"fun\", \"beautiful\",\n",
        "                     \"excellent\", \"poor\", \"boring\", \"awful\", \"terrible\",\n",
        "                     \"definitely\", \"perfect\", \"liked\", \"worse\", \"waste\",\n",
        "                     \"entertaining\", \"loved\", \"unfortunately\", \"amazing\",\n",
        "                     \"enjoyed\", \"favorite\", \"horrible\", \"brilliant\", \"highly\",\n",
        "                     \"simple\", \"annoying\", \"today\", \"hilarious\", \"enjoyable\",\n",
        "                     \"dull\", \"fantastic\", \"poorly\", \"fails\", \"disappointing\",\n",
        "                     \"disappointment\", \"not\", \"him\", \"her\", \"good\", \"time\",\n",
        "                     \"?\", \".\", \"!\", \"movie\", \"film\", \"action\", \"comedy\",\n",
        "                     \"drama\", \"family\")\n",
        "\n",
        "terms_feature_column = tf.feature_column.categorical_column_with_vocabulary_list(key=\"terms\", vocabulary_list=informative_terms)"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "eTiDwyorwd3P",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " \ub2e4\uc74c\uc73c\ub85c, `LinearClassifier`\ub97c \uc0dd\uc131\ud558\uace0 \ud559\uc2b5 \uc138\ud2b8\ub85c \ud559\uc2b5\uc2dc\ud0a8 \ud6c4 \ud3c9\uac00 \uc138\ud2b8\ub85c \ud3c9\uac00\ud569\ub2c8\ub2e4. \ucf54\ub4dc\ub97c \uc798 \uc77d\uc5b4\ubcf4\uace0 \uc2e4\ud589\ud558\uc5ec \uacb0\uacfc\ub97c \ud655\uc778\ud574 \ubcf4\uc138\uc694."
      ]
    },
    {
      "metadata": {
        "id": "HYKKpGLqLS8d",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "my_optimizer = tf.train.AdagradOptimizer(learning_rate=0.1)\n",
        "my_optimizer = tf.contrib.estimator.clip_gradients_by_norm(my_optimizer, 5.0)\n",
        "\n",
        "feature_columns = [ terms_feature_column ]\n",
        "\n",
        "\n",
        "classifier = tf.estimator.LinearClassifier(\n",
        "  feature_columns=feature_columns,\n",
        "  optimizer=my_optimizer,\n",
        ")\n",
        "\n",
        "classifier.train(\n",
        "  input_fn=lambda: _input_fn([train_path]),\n",
        "  steps=1000)\n",
        "\n",
        "evaluation_metrics = classifier.evaluate(\n",
        "  input_fn=lambda: _input_fn([train_path]),\n",
        "  steps=1000)\n",
        "print(\"Training set metrics:\")\n",
        "for m in evaluation_metrics:\n",
        "  print(m, evaluation_metrics[m])\n",
        "print(\"---\")\n",
        "\n",
        "evaluation_metrics = classifier.evaluate(\n",
        "  input_fn=lambda: _input_fn([test_path]),\n",
        "  steps=1000)\n",
        "\n",
        "print(\"Test set metrics:\")\n",
        "for m in evaluation_metrics:\n",
        "  print(m, evaluation_metrics[m])\n",
        "print(\"---\")"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "J0ubn9gULS8g",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " ## \uc791\uc5c5 2: \uc2ec\uce35\uc2e0\uacbd\ub9dd(DNN) \ubaa8\ub378 \uc0ac\uc6a9\n",
        "\n",
        "\uc704 \ubaa8\ub378\uc740 \uc120\ud615 \ubaa8\ub378\uc785\ub2c8\ub2e4. \ube44\uad50\uc801 \uc88b\uc740 \uc131\ub2a5\uc744 \ubc1c\ud718\ud558\uc9c0\ub9cc, DNN \ubaa8\ub378\ub85c \uc131\ub2a5\uc744 \ub354 \ub192\uc77c \uc218 \uc788\uc744\uae4c\uc694?\n",
        "\n",
        "`LinearClassifier`\ub97c [`DNNClassifier`](https://www.tensorflow.org/api_docs/python/tf/estimator/DNNClassifier)\ub85c \uad50\uccb4\ud574 \ubcf4\uaca0\uc2b5\ub2c8\ub2e4. \ub2e4\uc74c \uc140\uc744 \uc2e4\ud589\ud558\uace0 \uacb0\uacfc\ub97c \ud655\uc778\ud574 \ubcf4\uc138\uc694."
      ]
    },
    {
      "metadata": {
        "id": "jcgOPfEALS8h",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "##################### Here's what we changed ##################################\n",
        "classifier = tf.estimator.DNNClassifier(                                      #\n",
        "  feature_columns=[tf.feature_column.indicator_column(terms_feature_column)], #\n",
        "  hidden_units=[20,20],                                                       #\n",
        "  optimizer=my_optimizer,                                                     #\n",
        ")                                                                             #\n",
        "###############################################################################\n",
        "\n",
        "try:\n",
        "  classifier.train(\n",
        "    input_fn=lambda: _input_fn([train_path]),\n",
        "    steps=1000)\n",
        "\n",
        "  evaluation_metrics = classifier.evaluate(\n",
        "    input_fn=lambda: _input_fn([train_path]),\n",
        "    steps=1)\n",
        "  print(\"Training set metrics:\")\n",
        "  for m in evaluation_metrics:\n",
        "    print(m, evaluation_metrics[m])\n",
        "  print(\"---\")\n",
        "\n",
        "  evaluation_metrics = classifier.evaluate(\n",
        "    input_fn=lambda: _input_fn([test_path]),\n",
        "    steps=1)\n",
        "\n",
        "  print(\"Test set metrics:\")\n",
        "  for m in evaluation_metrics:\n",
        "    print(m, evaluation_metrics[m])\n",
        "  print(\"---\")\n",
        "except ValueError as err:\n",
        "  print(err)"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "cZz68luxLS8j",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " ## \uc791\uc5c5 3: DNN \ubaa8\ub378\uc5d0 \uc784\ubca0\ub529 \uc0ac\uc6a9\n",
        "\n",
        "\uc774 \uc791\uc5c5\uc5d0\uc11c\ub294 \uc784\ubca0\ub529 \uc5f4\uc744 \uc0ac\uc6a9\ud558\uc5ec DNN \ubaa8\ub378\uc744 \uad6c\ud604\ud569\ub2c8\ub2e4. \uc784\ubca0\ub529 \uc5f4\uc740 \ud76c\uc18c \ub370\uc774\ud130\ub97c \uc785\ub825\uc73c\ub85c \ucde8\ud558\uace0 \uc800\ucc28\uc6d0 \ubc00\uc9d1 \ubca1\ud130\ub97c \ucd9c\ub825\uc73c\ub85c \ubc18\ud658\ud569\ub2c8\ub2e4."
      ]
    },
    {
      "metadata": {
        "id": "AliRzhvJLS8k",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " **\ucc38\uace0:** *\ud76c\uc18c \ub370\uc774\ud130\ub85c \ubaa8\ub378\uc744 \ud559\uc2b5\uc2dc\ud0ac \ub54c embedding_column\uc740 \uc77c\ubc18\uc801\uc73c\ub85c \uc5f0\uc0b0 \ud6a8\uc728\uc774 \uac00\uc7a5 \ub192\uc740 \uc635\uc158\uc785\ub2c8\ub2e4. \uc774 \uc2e4\uc2b5 \ub05d\ubd80\ubd84\uc758 [\uc120\ud0dd \uc139\uc158](#scrollTo=XDMlGgRfKSVz)\uc5d0\uc11c `embedding_column`\uacfc `indicator_column`\uc758 \uad6c\ud604\uc0c1 \ucc28\uc774\uc810 \ubc0f \uc0c1\ub300\uc801\uc778 \uc7a5\ub2e8\uc810\uc744 \uc790\uc138\ud788 \uc54c\uc544\ubd05\ub2c8\ub2e4.*"
      ]
    },
    {
      "metadata": {
        "id": "F-as3PtALS8l",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " \uc544\ub798 \ucf54\ub4dc\uc5d0\uc11c \ub2e4\uc74c\uc744 \uc218\ud589\ud569\ub2c8\ub2e4.\n",
        "\n",
        "* \ub370\uc774\ud130\ub97c 2\uac1c\uc758 \ucc28\uc6d0\uc73c\ub85c \ud22c\uc601\ud558\ub294 `embedding_column`\uc744 \uc0ac\uc6a9\ud558\uc5ec \ubaa8\ub378\uc758 \ud2b9\uc131 \uc5f4\uc744 \uc815\uc758\ud569\ub2c8\ub2e4. `embedding_column`\uc758 \ud568\uc218 \uc2dc\uadf8\ub2c8\ucc98\uc5d0 \ub300\ud55c \uc790\uc138\ud55c \ub0b4\uc6a9\uc740 [TF \ubb38\uc11c](https://www.tensorflow.org/api_docs/python/tf/feature_column/embedding_column)\ub97c \ucc38\uc870\ud558\uc138\uc694.\n",
        "* \ub2e4\uc74c\uacfc \uac19\uc740 \uc0ac\uc591\uc73c\ub85c `DNNClassifier`\ub97c \uc815\uc758\ud569\ub2c8\ub2e4.\n",
        "  * \uac01\uac01 20\uac1c \uc720\ub2db\uc744 \ud3ec\ud568\ud558\ub294 \ud788\ub4e0 \ub808\uc774\uc5b4 2\uac1c\n",
        "  * Adagrad \ucd5c\uc801\ud654, \ud559\uc2b5\ub960 0.1\n",
        "  * `gradient_clip_norm`\uc744 5.0\uc73c\ub85c \uc9c0\uc815"
      ]
    },
    {
      "metadata": {
        "id": "UlPZ-Q9bLS8m",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " **\ucc38\uace0:** *\uc2e4\ubb34\uc5d0\uc11c\ub294 2\ubcf4\ub2e4 \ub192\uc740 50 \ub610\ub294 100\ucc28\uc6d0\uc73c\ub85c \ud22c\uc601\ud558\uac8c \ub429\ub2c8\ub2e4. \uadf8\ub7ec\ub098 \uc5ec\uae30\uc5d0\uc11c\ub294 \uc2dc\uac01\ud654\ud558\uae30 \uc27d\ub3c4\ub85d 2\ucc28\uc6d0\ub9cc \uc0ac\uc6a9\ud569\ub2c8\ub2e4.*"
      ]
    },
    {
      "metadata": {
        "id": "mNCLhxsXyOIS",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " ### \ud78c\ud2b8"
      ]
    },
    {
      "metadata": {
        "id": "L67xYD7hLS8m",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "# Here's a example code snippet you might use to define the feature columns:\n",
        "\n",
        "terms_embedding_column = tf.feature_column.embedding_column(terms_feature_column, dimension=2)\n",
        "feature_columns = [ terms_embedding_column ]"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "iv1UBsJxyV37",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " ### \uc544\ub798 \ucf54\ub4dc \uc644\uc131\ud558\uae30"
      ]
    },
    {
      "metadata": {
        "id": "5PG_yhNGLS8u",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "########################## YOUR CODE HERE ######################################\n",
        "terms_embedding_column = # Define the embedding column\n",
        "feature_columns = # Define the feature columns\n",
        "\n",
        "classifier = # Define the DNNClassifier\n",
        "################################################################################\n",
        "\n",
        "classifier.train(\n",
        "  input_fn=lambda: _input_fn([train_path]),\n",
        "  steps=1000)\n",
        "\n",
        "evaluation_metrics = classifier.evaluate(\n",
        "  input_fn=lambda: _input_fn([train_path]),\n",
        "  steps=1000)\n",
        "print(\"Training set metrics:\")\n",
        "for m in evaluation_metrics:\n",
        "  print(m, evaluation_metrics[m])\n",
        "print(\"---\")\n",
        "\n",
        "evaluation_metrics = classifier.evaluate(\n",
        "  input_fn=lambda: _input_fn([test_path]),\n",
        "  steps=1000)\n",
        "\n",
        "print(\"Test set metrics:\")\n",
        "for m in evaluation_metrics:\n",
        "  print(m, evaluation_metrics[m])\n",
        "print(\"---\")"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "eQS5KQzBybTY",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " ### \ud574\uacb0 \ubc29\ubc95\n",
        "\n",
        "\ud574\uacb0 \ubc29\ubc95\uc744 \ubcf4\ub824\uba74 \uc544\ub798\ub97c \ud074\ub9ad\ud558\uc138\uc694."
      ]
    },
    {
      "metadata": {
        "id": "R5xOdYeQydi5",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "########################## SOLUTION CODE ########################################\n",
        "terms_embedding_column = tf.feature_column.embedding_column(terms_feature_column, dimension=2)\n",
        "feature_columns = [ terms_embedding_column ]\n",
        "\n",
        "my_optimizer = tf.train.AdagradOptimizer(learning_rate=0.1)\n",
        "my_optimizer = tf.contrib.estimator.clip_gradients_by_norm(my_optimizer, 5.0)\n",
        "\n",
        "classifier = tf.estimator.DNNClassifier(\n",
        "  feature_columns=feature_columns,\n",
        "  hidden_units=[20,20],\n",
        "  optimizer=my_optimizer\n",
        ")\n",
        "#################################################################################\n",
        "\n",
        "classifier.train(\n",
        "  input_fn=lambda: _input_fn([train_path]),\n",
        "  steps=1000)\n",
        "\n",
        "evaluation_metrics = classifier.evaluate(\n",
        "  input_fn=lambda: _input_fn([train_path]),\n",
        "  steps=1000)\n",
        "print(\"Training set metrics:\")\n",
        "for m in evaluation_metrics:\n",
        "  print(m, evaluation_metrics[m])\n",
        "print(\"---\")\n",
        "\n",
        "evaluation_metrics = classifier.evaluate(\n",
        "  input_fn=lambda: _input_fn([test_path]),\n",
        "  steps=1000)\n",
        "\n",
        "print(\"Test set metrics:\")\n",
        "for m in evaluation_metrics:\n",
        "  print(m, evaluation_metrics[m])\n",
        "print(\"---\")"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "aiHnnVtzLS8w",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " ## \uc791\uc5c5 4: \uc784\ubca0\ub529\uc774 \uc2e4\uc81c\ub85c \uc801\uc6a9\ub418\ub294\uc9c0 \ud655\uc778\n",
        "\n",
        "\uc704 \ubaa8\ub378\uc5d0\uc11c \uc0ac\uc6a9\ud55c `embedding_column`\uc740 \uc81c\ub300\ub85c \uc791\ub3d9\ud558\ub294 \uac83 \uac19\uc9c0\ub9cc, \ub0b4\ubd80\uc801\uc73c\ub85c\ub294 \uc5b4\ub5bb\uac8c \uc0ac\uc6a9\ub418\ub294\uc9c0 \uc54c \uc218\uac00 \uc5c6\uc2b5\ub2c8\ub2e4. \ubaa8\ub378\uc5d0\uc11c \ub0b4\ubd80\uc801\uc73c\ub85c \uc784\ubca0\ub529\uc744 \uc2e4\uc81c\ub85c \uc0ac\uc6a9\ud558\ub294\uc9c0 \ud655\uc778\ud558\ub824\uba74 \uc5b4\ub5bb\uac8c \ud574\uc57c \ud560\uae4c\uc694?\n",
        "\n",
        "\uc6b0\uc120 \ubaa8\ub378\uc758 \ud150\uc11c\ub97c \uc0b4\ud3b4\ubcf4\uaca0\uc2b5\ub2c8\ub2e4."
      ]
    },
    {
      "metadata": {
        "id": "h1jNgLdQLS8w",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "classifier.get_variable_names()"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Sl4-VctMLS8z",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " \uc774\uc81c `'dnn/input_from_feature_columns/input_layer/terms_embedding/...'`\uc774\ub77c\ub294 \uc784\ubca0\ub529 \ub808\uc774\uc5b4\uac00 \uc788\uc74c\uc744 \ud655\uc778\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4. \uc5ec\uae30\uc5d0\uc11c \ud765\ubbf8\ub85c\uc6b4 \uc810\uc740 \uc774 \ub808\uc774\uc5b4\ub294 \uc5ec\ud0c0 \ud788\ub4e0 \ub808\uc774\uc5b4\uc640 \ub9c8\ucc2c\uac00\uc9c0\ub85c \ubaa8\ub378\uc758 \ub2e4\ub978 \ubd80\ubd84\uacfc \ud568\uaed8 \ub3d9\uc2dc\uc5d0 \ud559\uc2b5\ub41c\ub2e4\ub294 \uc810\uc785\ub2c8\ub2e4.\n",
        "\n",
        "\uc784\ubca0\ub529 \ub808\uc774\uc5b4\uac00 \uc62c\ubc14\ub978 \ud615\ud0dc\ub85c \ub418\uc5b4 \uc788\uc744\uae4c\uc694? \ub2e4\uc74c \ucf54\ub4dc\ub97c \uc2e4\ud589\ud558\uc5ec \uc54c\uc544\ubcf4\uc138\uc694."
      ]
    },
    {
      "metadata": {
        "id": "JNFxyQUiLS80",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " **\ucc38\uace0:** *\uc5ec\uae30\uc5d0\uc11c \uc0ac\uc6a9\ud558\ub294 \uc784\ubca0\ub529\uc740 50\ucc28\uc6d0 \ubca1\ud130\ub97c 2\ucc28\uc6d0\uc73c\ub85c \ud22c\uc601\ud558\ub294 \ud589\ub82c\uc785\ub2c8\ub2e4.*"
      ]
    },
    {
      "metadata": {
        "id": "1xMbpcEjLS80",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "classifier.get_variable_value('dnn/input_from_feature_columns/input_layer/terms_embedding/embedding_weights').shape"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "MnLCIogjLS82",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " \uc7a0\uc2dc \ub3d9\uc548 \ub2e4\uc591\ud55c \ub808\uc774\uc5b4\uc640 \ud615\ud0dc\ub97c \uc9c1\uc811 \ud655\uc778\ud558\uc5ec \ubaa8\ub4e0 \uc694\uc18c\uac00 \uc608\uc0c1\ub300\ub85c \uc5f0\uacb0\ub418\uc5b4 \uc788\ub294\uc9c0 \ud655\uc778\ud574 \ubcf4\uc138\uc694."
      ]
    },
    {
      "metadata": {
        "id": "rkKAaRWDLS83",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " ## \uc791\uc5c5 5: \uc784\ubca0\ub529 \uc870\uc0ac\n",
        "\n",
        "\uc774\uc81c \uc2e4\uc81c \uc784\ubca0\ub529 \uacf5\uac04\uc744 \uc870\uc0ac\ud558\uc5ec \uac01 \ub2e8\uc5b4\uac00 \uacb0\uad6d \uc5b4\ub290 \uc704\uce58\uc5d0 \ubc30\uce58\ub418\uc5c8\ub294\uc9c0 \ud655\uc778\ud574 \ubcf4\uaca0\uc2b5\ub2c8\ub2e4. \ub2e4\uc74c\uc744 \uc218\ud589\ud558\uc138\uc694.\n",
        "1. \ub2e4\uc74c \ucf54\ub4dc\ub97c \uc2e4\ud589\ud558\uc5ec **\uc791\uc5c5 3**\uc5d0\uc11c \ud559\uc2b5\uc2dc\ud0a8 \uc784\ubca0\ub529\uc744 \ud655\uc778\ud569\ub2c8\ub2e4. \uacb0\uacfc\uac00 \uc608\uc0c1\uacfc \uc77c\uce58\ud558\ub098\uc694?\n",
        "\n",
        "2. **\uc791\uc5c5 3**\uc758 \ucf54\ub4dc\ub97c \uc7ac\uc2e4\ud589\ud558\uc5ec \ubaa8\ub378\uc744 \ub2e4\uc2dc \ud559\uc2b5\uc2dc\ud0a8 \ud6c4 \uc544\ub798\uc758 \uc784\ubca0\ub529 \uc2dc\uac01\ud654\ub97c \ub2e4\uc2dc \uc2e4\ud589\ud569\ub2c8\ub2e4. \ubb34\uc5c7\uc774 \uadf8\ub300\ub85c\uc778\uac00\uc694? \ubb34\uc5c7\uc774 \ub2ec\ub77c\uc84c\ub098\uc694?\n",
        "\n",
        "3. \ub9c8\uc9c0\ub9c9\uc73c\ub85c 10\ub2e8\uacc4\ub9cc \uc0ac\uc6a9\ud558\uc5ec \ubaa8\ub378\uc744 \ub2e4\uc2dc \ud559\uc2b5\uc2dc\ud0b5\ub2c8\ub2e4. \uc774\ub807\uac8c \ud558\uba74 \ub9e4\uc6b0 \uc5f4\uc545\ud55c \ubaa8\ub378\uc774 \ub9cc\ub4e4\uc5b4\uc9d1\ub2c8\ub2e4. \uc544\ub798\uc758 \uc784\ubca0\ub529 \uc2dc\uac01\ud654\ub97c \ub2e4\uc2dc \uc2e4\ud589\ud569\ub2c8\ub2e4. \uc774\uc81c \uacb0\uacfc\uac00 \uc5b4\ub5a0\ud55c\uac00\uc694? \uc774\uc720\ub294 \ubb34\uc5c7\uc77c\uae4c\uc694?"
      ]
    },
    {
      "metadata": {
        "id": "s4NNu7KqLS84",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "embedding_matrix = classifier.get_variable_value('dnn/input_from_feature_columns/input_layer/terms_embedding/embedding_weights')\n",
        "\n",
        "for term_index in range(len(informative_terms)):\n",
        "  # Create a one-hot encoding for our term.  It has 0s everywhere, except for\n",
        "  # a single 1 in the coordinate that corresponds to that term.\n",
        "  term_vector = np.zeros(len(informative_terms))\n",
        "  term_vector[term_index] = 1\n",
        "  # We'll now project that one-hot vector into the embedding space.\n",
        "  embedding_xy = np.matmul(term_vector, embedding_matrix)\n",
        "  plt.text(embedding_xy[0],\n",
        "           embedding_xy[1],\n",
        "           informative_terms[term_index])\n",
        "\n",
        "# Do a little setup to make sure the plot displays nicely.\n",
        "plt.rcParams[\"figure.figsize\"] = (15, 15)\n",
        "plt.xlim(1.2 * embedding_matrix.min(), 1.2 * embedding_matrix.max())\n",
        "plt.ylim(1.2 * embedding_matrix.min(), 1.2 * embedding_matrix.max())\n",
        "plt.show() "
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "pUb3L7pqLS86",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " ## \uc791\uc5c5 6: \ubaa8\ub378 \uc131\ub2a5 \uac1c\uc120 \uc2dc\ub3c4\n",
        "\n",
        "\ubaa8\ub378\uc744 \ub2e4\ub4ec\uc5b4 \uc131\ub2a5\uc744 \ub192\uc77c \uc218 \uc788\ub294\uc9c0 \ud655\uc778\ud574 \ubcf4\uc138\uc694. \ub2e4\uc74c\uacfc \uac19\uc740 \ubc29\ubc95\uc744 \uc2dc\ub3c4\ud574 \ubcfc \uc218 \uc788\uc2b5\ub2c8\ub2e4.\n",
        "\n",
        "* **\ucd08\ub9e4\uac1c\ubcc0\uc218 \ubcc0\uacbd** \ub610\ub294 Adam \ub4f1\uc758 **\ub2e4\ub978 \uc635\ud2f0\ub9c8\uc774\uc800 \uc0ac\uc6a9**. \uc774 \uc804\ub7b5\uc73c\ub85c \ud5a5\uc0c1\ub418\ub294 \uc815\ud655\uc131\uc740 1~2%\uc5d0 \ubd88\uacfc\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4.\n",
        "* **`informative_terms`\uc5d0 \ub354 \ub9ce\uc740 \ub2e8\uc5b4 \ucd94\uac00.** \uc774 \ub370\uc774\ud130 \uc138\ud2b8\uc758 30,716\uac1c \ub2e8\uc5b4\ub97c \ubaa8\ub450 \ud3ec\ud568\ud558\ub294 \uc804\uccb4 \uc5b4\ud718 \ud30c\uc77c\uc740 https://download.mlcc.google.com/mledu-datasets/sparse-data-embedding/terms.txt \uc785\ub2c8\ub2e4. \uc774 \uc5b4\ud718 \ud30c\uc77c\uc5d0\uc11c \ub2e8\uc5b4\ub97c \ub354 \ucd94\ucd9c\ud560 \uc218\ub3c4 \uc788\uace0, `categorical_column_with_vocabulary_file` \ud2b9\uc131 \uc5f4\uc744 \ud1b5\ud574 \uc804\uccb4 \uc5b4\ud718\ub97c \uc0ac\uc6a9\ud560 \uc218\ub3c4 \uc788\uc2b5\ub2c8\ub2e4."
      ]
    },
    {
      "metadata": {
        "id": "6-b3BqXvLS86",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "# Download the vocabulary file.\n",
        "terms_url = 'https://download.mlcc.google.com/mledu-datasets/sparse-data-embedding/terms.txt'\n",
        "terms_path = tf.keras.utils.get_file(terms_url.split('/')[-1], terms_url)"      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "0jbJlwW5LS8-",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "# Create a feature column from \"terms\", using a full vocabulary file.\n",
        "informative_terms = None\n",
        "with io.open(terms_path, 'r', encoding='utf8') as f:\n",
        "  # Convert it to a set first to remove duplicates.\n",
        "  informative_terms = list(set(f.read().split()))\n",
        "  \n",
        "terms_feature_column = tf.feature_column.categorical_column_with_vocabulary_list(key=\"terms\", \n",
        "                                                                                 vocabulary_list=informative_terms)\n",
        "\n",
        "terms_embedding_column = tf.feature_column.embedding_column(terms_feature_column, dimension=2)\n",
        "feature_columns = [ terms_embedding_column ]\n",
        "\n",
        "my_optimizer = tf.train.AdagradOptimizer(learning_rate=0.1)\n",
        "my_optimizer = tf.contrib.estimator.clip_gradients_by_norm(my_optimizer, 5.0)\n",
        "\n",
        "classifier = tf.estimator.DNNClassifier(\n",
        "  feature_columns=feature_columns,\n",
        "  hidden_units=[10,10],\n",
        "  optimizer=my_optimizer\n",
        ")\n",
        "\n",
        "classifier.train(\n",
        "  input_fn=lambda: _input_fn([train_path]),\n",
        "  steps=1000)\n",
        "\n",
        "evaluation_metrics = classifier.evaluate(\n",
        "  input_fn=lambda: _input_fn([train_path]),\n",
        "  steps=1000)\n",
        "print(\"Training set metrics:\")\n",
        "for m in evaluation_metrics:\n",
        "  print(m, evaluation_metrics[m])\n",
        "print(\"---\")\n",
        "\n",
        "evaluation_metrics = classifier.evaluate(\n",
        "  input_fn=lambda: _input_fn([test_path]),\n",
        "  steps=1000)\n",
        "\n",
        "print(\"Test set metrics:\")\n",
        "for m in evaluation_metrics:\n",
        "  print(m, evaluation_metrics[m])\n",
        "print(\"---\")"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ew3kwGM-LS9B",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " ## \ub9fa\uc74c\ub9d0\n",
        "\n",
        "\uc784\ubca0\ub529\uc744 \uc0ac\uc6a9\ud55c DNN \uc194\ub8e8\uc158\uc774 \uc6d0\ub798\uc758 \uc120\ud615 \ubaa8\ub378\ubcf4\ub2e4 \uc6b0\uc218\ud560 \uc218 \uc788\uc9c0\ub9cc, \uc120\ud615 \ubaa8\ub378\ub3c4 \uc131\ub2a5\uc774 \uadf8\ub2e4\uc9c0 \ub098\uc058\uc9c0 \uc54a\uc558\uc73c\uba70 \ud559\uc2b5 \uc18d\ub3c4\ub294 \uc0c1\ub2f9\ud788 \ub354 \ube68\ub790\uc2b5\ub2c8\ub2e4. \uc120\ud615 \ubaa8\ub378\uc758 \ud559\uc2b5 \uc18d\ub3c4\uac00 \ub354 \ube60\ub978 \uc774\uc720\ub294 \uc5c5\ub370\uc774\ud2b8\ud560 \ub9e4\uac1c\ubcc0\uc218 \ub610\ub294 \uc5ed\uc804\ud30c\ud560 \ub808\uc774\uc5b4\uc758 \uc218\uac00 \ub354 \uc801\uae30 \ub54c\ubb38\uc785\ub2c8\ub2e4.\n",
        "\n",
        "\uc751\uc6a9 \ubd84\uc57c\uc5d0 \ub530\ub77c\uc11c\ub294 \uc120\ud615 \ubaa8\ub378\uc758 \ube60\ub978 \uc18d\ub3c4\uac00 \ud070 \uc7a5\uc810\uc774 \ub420 \uc218 \uc788\uace0, \uc120\ud615 \ubaa8\ub378\ub3c4 \ud488\uc9c8 \uba74\uc5d0\uc11c \ucda9\ubd84\ud558\uace0\ub3c4 \ub0a8\uc744 \uc218 \uc788\uc2b5\ub2c8\ub2e4. \ub2e4\ub978 \ubd84\uc57c\uc5d0\uc11c\ub294 DNN\uc774 \uc81c\uacf5\ud558\ub294 \ucd94\uac00\uc801\uc778 \ubaa8\ub378 \ubcf5\uc7a1\uc131\uacfc \uc6a9\ub7c9\uc774 \ub354 \uc911\uc694\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4. \ubaa8\ub378 \uc544\ud0a4\ud14d\ucc98\ub97c \uc815\uc758\ud560 \ub54c\ub294 \uc5b4\ub5a0\ud55c \ubaa8\ub378\uc774 \uc801\ud569\ud55c\uc9c0 \ud310\ub2e8\ud560 \uc218 \uc788\ub3c4\ub85d \ubb38\uc81c\ub97c \ucda9\ubd84\ud788 \ud0d0\uad6c\ud574\uc57c \ud569\ub2c8\ub2e4."
      ]
    },
    {
      "metadata": {
        "id": "9MquXy9zLS9B",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " ### *\uc120\ud0dd \ud1a0\uc758:* `embedding_column`\uacfc `indicator_column`\uc758 \uc0c1\ub300\uc801\uc778 \uc7a5\ub2e8\uc810\n",
        "\n",
        "\uac1c\ub150\uc801\uc73c\ub85c, `LinearClassifier` \ub610\ub294 `DNNClassifier`\ub97c \ud559\uc2b5\uc2dc\ud0ac \ub54c \ud76c\uc18c \uc5f4\uc744 \uc0ac\uc6a9\ud558\ub824\uba74 \uc5b4\ub311\ud130\uac00 \ud544\uc694\ud569\ub2c8\ub2e4. TF\ub294 `embedding_column` \ub610\ub294 `indicator_column`\uc774\ub77c\ub294 \ub450 \uac00\uc9c0 \uc635\uc158\uc744 \uc81c\uacf5\ud569\ub2c8\ub2e4.\n",
        "\n",
        "**\uc791\uc5c5 1**\uacfc \uac19\uc774 LinearClassifier\ub97c \ud559\uc2b5\uc2dc\ud0ac \ub54c\ub294 \ub0b4\ubd80\uc801\uc73c\ub85c `embedding_column`\uc774 \uc0ac\uc6a9\ub429\ub2c8\ub2e4. **\uc791\uc5c5 2**\uc5d0\uc11c \ud655\uc778\ud55c \ubc14\uc640 \uac19\uc774 `DNNClassifier`\ub97c \ud559\uc2b5\uc2dc\ud0ac \ub54c\ub294 `embedding_column` \ub610\ub294 `indicator_column` \uc911 \ud558\ub098\ub97c \uba85\uc2dc\uc801\uc73c\ub85c \uc120\ud0dd\ud574\uc57c \ud569\ub2c8\ub2e4. \uc774 \uc139\uc158\uc5d0\uc11c\ub294 \uac04\ub2e8\ud55c \uc608\ub97c \uc0b4\ud3b4\ubcf4\uba74\uc11c \ub458 \uc0ac\uc774\uc758 \ucc28\uc774\uc810 \ubc0f \uc0c1\ub300\uc801\uc778 \uc7a5\ub2e8\uc810\uc5d0 \ub300\ud574 \uc124\uba85\ud569\ub2c8\ub2e4."
      ]
    },
    {
      "metadata": {
        "id": "M_3XuZ_LLS9C",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " `\"great\"`, `\"beautiful\"`, `\"excellent\"` \uac12\uc744 \ud3ec\ud568\ud558\ub294 \ud76c\uc18c \ub370\uc774\ud130\uac00 \uc788\ub2e4\uace0 \uac00\uc815\ud574 \ubcf4\uaca0\uc2b5\ub2c8\ub2e4. \uc5ec\uae30\uc5d0\uc11c \uc0ac\uc6a9\ud558\ub294 \uc5b4\ud718 \ud06c\uae30\ub294 $V = 50$\uc774\ubbc0\ub85c \uccab \ubc88\uc9f8 \ub808\uc774\uc5b4\uc758 \uac01 \uc720\ub2db(\ub274\ub7f0)\uc740 50\uac1c\uc758 \uac00\uc911\uce58\ub97c \uac16\uc2b5\ub2c8\ub2e4. \ud76c\uc18c \uc785\ub825\uc758 \ub2e8\uc5b4 \uc218\ub294 $s$\ub85c \ub098\ud0c0\ub0c5\ub2c8\ub2e4. \ub530\ub77c\uc11c \uc774 \ud76c\uc18c \ub370\uc774\ud130 \uc608\uc5d0\uc11c $s = 3$\uc785\ub2c8\ub2e4. \uac00\ub2a5\ud55c \uac12\uc774 $V$\uac1c\uc778 \uc785\ub825 \ub808\uc774\uc5b4\uc5d0 \ub300\ud574 $d$\uac1c \uc720\ub2db\uc744 \uac16\ub294 \ud788\ub4e0 \ub808\uc774\uc5b4\uac00 \uacc4\uc0b0\ud574\uc57c \ud558\ub294 \ubca1\ud130 \ud589\ub82c \uacf1\uc148\uc740 $(1 \\times V) * (V \\times d)$\uc785\ub2c8\ub2e4. \uc5f0\uc0b0 \ube44\uc6a9\uc740 $O(V * d)$\uc785\ub2c8\ub2e4. \uc774 \ube44\uc6a9\uc740 \ud574\ub2f9 \ud788\ub4e0 \ub808\uc774\uc5b4\uc758 \uac00\uc911\uce58 \uc218\uc5d0 \ube44\ub840\ud558\uba70 $s$\uc640\ub294 \ubb34\uad00\ud569\ub2c8\ub2e4.\n",
        "\n",
        "[`indicator_column`](https://www.tensorflow.org/api_docs/python/tf/feature_column/indicator_column)\uc744 \uc0ac\uc6a9\ud558\uc5ec \uc785\ub825\uc5d0 \uc6d0-\ud56b \uc778\ucf54\ub529(\uae38\uc774\uac00 $V$\uc774\uace0 \uc874\uc7ac\ud558\ub294 \ub2e8\uc5b4\uc5d0 \ub300\ud574 1, \uadf8\ub807\uc9c0 \uc54a\uc740 \ub2e8\uc5b4\uc5d0 \ub300\ud574 0\uc744 \ud3ec\ud568\ud558\ub294 \ubd80\uc6b8 \ubca1\ud130)\uc744 \uc801\uc6a9\ud558\uba74 \uacf1\uc148\uacfc \ub367\uc148\uc758 \ub9ce\uc740 \ud56d\uc5d0 0\uc774 \ub4e4\uc5b4\uac11\ub2c8\ub2e4."
      ]
    },
    {
      "metadata": {
        "id": "I7mR4Wa2LS9C",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " \ud06c\uae30\uac00 $d$\uc778 [`embedding_column`](https://www.tensorflow.org/api_docs/python/tf/feature_column/embedding_column)\uc744 \uc0ac\uc6a9\ud574\ub3c4 \uc815\ud655\ud788 \uac19\uc740 \uacb0\uacfc\ub97c \uc5bb\uc744 \uc218 \uc788\uc73c\uba70, \ub2e4\uc74c\uacfc \uac19\uc774 \uc785\ub825 \uc608\uc778 `\"great\"`, `\"beautiful\"`, `\"excellent\"`\uc5d0 \uc788\ub294 3\uac1c \ud2b9\uc131\uc5d0 \ub300\ud55c \uc784\ubca0\ub529\ub9cc \uc870\ud68c\ud558\uc5ec \ud569\uc0b0\ud569\ub2c8\ub2e4. $(1 \\times d) + (1 \\times d) + (1 \\times d)$. \uc874\uc7ac\ud558\uc9c0 \uc54a\ub294 \ud2b9\uc131\uc5d0 \ub300\ud55c \uac00\uc911\uce58\ub294 \ubca1\ud130 \ud589\ub82c \uacf1\uc148\uc5d0\uc11c 0\uc774 \uacf1\ud574\uc9c0\ubbc0\ub85c \uacb0\uacfc\uc5d0 \uc804\ud600 \uae30\uc5ec\ud558\uc9c0 \uc54a\uc2b5\ub2c8\ub2e4. \uc874\uc7ac\ud558\ub294 \ud2b9\uc131\uc5d0 \ub300\ud55c \uac00\uc911\uce58\ub294 \ubca1\ud130 \ud589\ub82c \uacf1\uc148\uc5d0\uc11c 1\uc774 \uacf1\ud574\uc9d1\ub2c8\ub2e4. \ub530\ub77c\uc11c \uc784\ubca0\ub529 \uc870\ud68c\ub97c \ud1b5\ud574 \ud68d\ub4dd\ud55c \uac00\uc911\uce58\ub97c \ub354\ud55c \uacb0\uacfc\ub294 \ubca1\ud130 \ud589\ub82c \uacf1\uc148\uc758 \uacb0\uacfc\uc640 \ub3d9\uc77c\ud569\ub2c8\ub2e4.\n",
        "\n",
        "\uc784\ubca0\ub529\uc744 \uc0ac\uc6a9\ud560 \ub54c \uc784\ubca0\ub529 \uc870\ud68c\uc5d0 \ud544\uc694\ud55c \uc5f0\uc0b0 \ube44\uc6a9\uc740 $O(s * d)$\ub85c\uc11c, $s$\uac00 $V$\ubcf4\ub2e4 \ud6e8\uc52c \uc791\uc740 \uacbd\uc6b0 \ud76c\uc18c \ub370\uc774\ud130\uc758 `indicator_column`\uc5d0 \ub300\ud55c \ube44\uc6a9\uc778 $O(V * d)$\ubcf4\ub2e4 \ud6e8\uc52c \ub354 \ud6a8\uc728\uc801\uc785\ub2c8\ub2e4. \uc774\ub7ec\ud55c \uc784\ubca0\ub529\uc740 \ud559\uc2b5 \uc911\uc778 \ub300\uc0c1\uc774\ubbc0\ub85c \uc5b4\ub5a0\ud55c \ud559\uc2b5 \ubc18\ubcf5\uc5d0\uc11c\ub3c4 \uc9c0\uae08 \uc870\ud68c \uc911\uc778 \ud604\uc7ac \uac00\uc911\uce58\uac00 \ub429\ub2c8\ub2e4."
      ]
    },
    {
      "metadata": {
        "id": "etZ9qf0kLS9D",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " **\uc791\uc5c5 3**\uc5d0\uc11c \ud655\uc778\ud55c \uac83\ucc98\ub7fc, `DNNClassifier`\ub97c \ud559\uc2b5\uc2dc\ud0ac \ub54c `embedding_column`\uc744 \uc0ac\uc6a9\ud558\uba74 \ubaa8\ub378\uc5d0\uc11c \ud2b9\uc131\uc758 \uc800\ucc28\uc6d0 \ud45c\ud604\uc744 \ud559\uc2b5\ud558\ub294\ub370, \uc774\ub54c \uc720\uc0ac\uc131 \uce21\uc815\ud56d\ubaa9\uc740 \ud2b9\uc131 \ubca1\ud130 \uac04\uc758 \ub0b4\uc801\uac12\uc73c\ub85c \uc815\uc758\ud569\ub2c8\ub2e4. \uc774 \uc608\uc5d0\uc11c\ub294 \uc601\ud654 \ub9ac\ubdf0\ub77c\ub294 \ub9e5\ub77d\uc5d0\uc11c \ube44\uc2b7\ud558\uac8c \uc0ac\uc6a9\ub41c `\"great\"`, `\"excellent\"` \ub4f1\uc758 \ub2e8\uc5b4\ub294 \uc784\ubca0\ub529 \uacf5\uac04\uc5d0\uc11c \uc11c\ub85c \uac00\uae4c\uc774 \uc704\uce58\ud558\ubbc0\ub85c \ub0b4\uc801\uac12\uc774 \ud06c\uace0, `\"great\"`\uc640 `\"bad\"` \uac19\uc774 \uc0c1\uc774\ud55c \ub2e8\uc5b4\ub294 \uc784\ubca0\ub529 \uacf5\uac04\uc5d0\uc11c \uba40\ub9ac \ub5a8\uc5b4\uc838 \uc788\uc73c\ubbc0\ub85c \ub0b4\uc801\uac12\uc774 \uc791\uc2b5\ub2c8\ub2e4."
      ]
    }
  ]
}
